Investigation of Pre-Trained Bidirectional Encoder Representations from Transformers Checkpoints for Indonesian Abstractive Text Summarization

Text summarization aims to reduce text by removing less useful information obtain quickly and precisely. In Indonesian abstractive summarization, the research mostly focuses on multi-document which methods will not work optimally in single-document summarization. As public datasets works English are focusing this study emphasized Abstractive studies frequently use Bidirectional Encoder Representations from Transformers (BERT), since BERT checkpoint is available, it was employed study. This investigated of IndoSum dataset using BERTSum model. The investigation proceeded various combinations model encoders, embedding sizes, decoders. Evaluation results showed that models with more size used Generative Pre-Training (GPT)-like decoder could improve Recall-Oriented Understudy for Gisting (ROUGE) score BERTScore results.